The caching of the _data dir between CI runs seems to have worked: the mnist data was not re-downloaded during the build step for #67.

LGTM, and will improve CI performance. Just one nit about educating people to spoof headers responsibly.

Unfortunately this is not working for me - I get files that apparently are not in gzip format. Maybe this is why the CI is still failing too?

Maybe this is why the CI is still failing too?

It looks like the CI failure was just related to execution timeout again (not related to downloading). The build artifact makes it seem like the cache is working properly (there would be print statements like "Downloading xyz" at the code cell if the data were being downloaded again).

The failure due to gzip is indeed strange - the cached files are likely compressed otherwise execution would be failing on the decompression step too, though the docs for iter_content do say that compressed files will be automatically be decompressed. I'm not sure that behavior is always consistent, otherwise I don't understand how the cache was originally built with the still-compressed files.

Some possible solutions that come to mind are:

1. Use response.raw instead of response.iter_content, which shouldn't auto-decompress the content
2. Wrap the local data loading in a try-except: for gzip.

I'm partial to the second option - it adds a minor amount of boilerplate but should work regardless whether the data was decompressed when it was downloaded.

Ah - another potential problem is that the download is simply failing :). Testing locally now I'm getting the 503 errors (instead of the 403 forbidden from before). There currently isn't a check of the response status so that should be updated as well.

Ultimately I think this PR puts the necessary infrastructure in place to cache the data, but we may also need to change the data source as the original website does not seem very reliable.

Sounds reasonable - thanks @rossbar !

Thank you @rossbar

Note / ToDo from the meeting: Adding a warning sign

cc @melissawm